chore(website): migrate to pnpm and enable minimum release age by B4nan · Pull Request #1845 · apify/crawlee-python

B4nan · 2026-04-14T14:38:30Z

Summary

Part of the org-wide supply-chain hardening + pnpm migration. Migrates the website/ subdirectory of crawlee-python from Yarn 4 to pnpm. Only website/ changes — the Python root (uv) is untouched. Adds @apify/* + @crawlee/* whitelist at the Renovate layer (minimumReleaseAge: "1 day" was already set).

Changes

website/package.json: renamed to crawlee-python-website, marked private: true; packageManager: pnpm@10.24.0; yarn / npx → pnpm in scripts
website/pnpm-workspace.yaml: minimumReleaseAge: 1440 with @apify/* + @crawlee/* excluded
website/.npmrc: node-linker=hoisted, link-workspace-packages=true, prefer-workspace-packages=true, public-hoist-pattern[]=*
website/docusaurus.config.js: rename future.experimental_faster → future.faster (Docusaurus 3.10 renamed the key; pnpm resolves 3.10 from the ^3.9.2 range)
Deleted website/yarn.lock and website/.yarnrc.yml; generated website/pnpm-lock.yaml
pyproject.toml poe tasks build-docs / run-docs: drop corepack enable && yarn && yarn build/start, use pnpm install --frozen-lockfile && pnpm build/start
New .github/actions/pnpm-install composite action (cached pnpm store, pattern from chore: move to pnpm from yarn apify-cli#1068)
manual_release_stable.yaml version_docs job: use pnpm/action-setup + pnpm install --frozen-lockfile instead of corepack enable && yarn install; npx docusaurus → pnpm exec docusaurus; also commits pnpm-lock.yaml alongside versioned docs
renovate.json: add internalChecksFilter: "strict" and @apify/* + @crawlee/* whitelist packageRule (minimumReleaseAge: "1 day" was already present)

🤖 Generated with Claude Code

codecov · 2026-04-14T14:42:46Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.35%. Comparing base (306dc46) to head (93bad29).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1845      +/-   ##
==========================================
- Coverage   92.42%   92.35%   -0.08%     
==========================================
  Files         157      157              
  Lines       10921    10921              
==========================================
- Hits        10094    10086       -8     
- Misses        827      835       +8

Flag	Coverage Δ
unit	`92.35% <ø> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Pijukatel

Wait, we need to add uv specific excludes newer, I will add soon

Pijukatel

Please add this to pyproject.toml

[tool.uv.exclude-newer-package]
apify-client = false 
apify-shared = false 
apify_fingerprint_datapoints = false

B4nan · 2026-04-15T14:18:20Z

Added in commit b4f9b59 — plus the same block in apify-client-python and apify-sdk-python. On the JS side, apify-client, apify, crawlee, and got-scraping were added to minimumReleaseAgeExclude in pnpm-workspace.yaml across the 11 JS repos.

vdusek

From Claude:

A couple of inconsistencies compared to the other two repos (apify-sdk-python #855, apify-client-python #724):

.gitignore — No change here, but apify-sdk-python replaces website/.yarn with website/.pnpm-store. Should this repo also update .gitignore?
pyproject.toml poe tasks — build-docs / run-docs use pnpm install --frozen-lockfile && pnpm build, while the other two repos use pnpm install && uv run pnpm build (no --frozen-lockfile, wrapped with uv run). Is this intentional?
update-docs-theme poe task — The other two repos update this task from corepack enable && yarn up to pnpm update. Is this task missing here, or was it intentionally skipped?

B4nan · 2026-04-16T07:56:29Z

We have some docs theme in crawlee? Sounds like claude hallucinates here and compares repos that are part of apify-docs with crawlee.

vdusek · 2026-04-16T07:58:21Z

Yeah, 3 is hallucination, 1 seems legit, and 2 - maybe?

Migrates the website/ subdirectory from Yarn 4 to pnpm. The Python root (uv) is untouched. Adds @apify/* + @crawlee/* whitelist at the Renovate layer (minimumReleaseAge was already present in renovate.json). - website/package.json: rename to crawlee-python-website + private: true, packageManager -> pnpm@10.24.0, yarn/npx -> pnpm - website/pnpm-workspace.yaml: minimumReleaseAge 1440 with @apify/* + @crawlee/* excluded - website/.npmrc: hoisted linker + workspace linking flags - website/docusaurus.config.js: rename future.experimental_faster -> future.faster (Docusaurus 3.10 renamed the key) - Delete website/yarn.lock and website/.yarnrc.yml - pyproject.toml poe tasks build-docs/run-docs: drop corepack + yarn, use pnpm install --frozen-lockfile + pnpm build/start - New .github/actions/pnpm-install composite action (pattern from apify/apify-cli#1068) - manual_release_stable.yaml version_docs job: pnpm/action-setup + pnpm install --frozen-lockfile instead of corepack + yarn; npx docusaurus -> pnpm exec docusaurus; commit pnpm-lock.yaml - renovate.json: add internalChecksFilter "strict" and @apify/* + @crawlee/* whitelist (minimumReleaseAge "1 day" was already set) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replaces the local .github/actions/pnpm-install composite copy with the shared one from apify/workflows@main. Identical behavior, less duplication.

apify-client, apify-shared, and apify_fingerprint_datapoints are internal; their releases should install immediately (no 24h wait). Mirrors the @apify/* + @crawlee/* whitelist in pnpm-workspace.yaml on the JS side.

Silences npm warnings about unknown options like node-linker; pnpm reads the same keys from pnpm-workspace.yaml in camelCase form.

Block accidental npm/yarn install — npm 10.5+ and pnpm 10.x both honor devEngines.packageManager and refuse to run when it doesn't match.

- Replace website/.yarn with website/.pnpm-store in .gitignore - Add crawlee to uv exclude-newer-package whitelist - Add update-docs-theme poe task (consistent with sdk/client repos) - Use 'pnpm install && uv run pnpm build' in poe tasks (no --frozen-lockfile)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

devEngines.packageManager breaks pnpm commands that delegate to npm (pnpm version, pnpm pkg set, etc). Use the battle-tested only-allow approach instead (same as Vite, Vue, Astro).

github-actions bot assigned B4nan Apr 14, 2026

github-actions bot added this to the 138th sprint - Tooling team milestone Apr 14, 2026

github-actions bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Apr 14, 2026

B4nan added the adhoc Ad-hoc unplanned task added during the sprint. label Apr 14, 2026

B4nan marked this pull request as ready for review April 15, 2026 13:41

Pijukatel approved these changes Apr 15, 2026

View reviewed changes

Pijukatel self-requested a review April 15, 2026 14:02

Pijukatel requested changes Apr 15, 2026

View reviewed changes

Pijukatel reviewed Apr 15, 2026

View reviewed changes

Pijukatel approved these changes Apr 15, 2026

View reviewed changes

vdusek approved these changes Apr 16, 2026

View reviewed changes

vdusek self-requested a review April 16, 2026 07:46

vdusek reviewed Apr 16, 2026

View reviewed changes

vdusek self-requested a review April 16, 2026 07:56

B4nan and others added 7 commits April 16, 2026 10:12

chore: retrigger CI

b54df4b

ci: use shared apify/workflows/pnpm-install action

dca112c

Replaces the local .github/actions/pnpm-install composite copy with the shared one from apify/workflows@main. Identical behavior, less duplication.

chore: whitelist internal packages in uv exclude-newer

a9a38ea

apify-client, apify-shared, and apify_fingerprint_datapoints are internal; their releases should install immediately (no 24h wait). Mirrors the @apify/* + @crawlee/* whitelist in pnpm-workspace.yaml on the JS side.

chore: move pnpm config from .npmrc to pnpm-workspace.yaml

931a6ae

Silences npm warnings about unknown options like node-linker; pnpm reads the same keys from pnpm-workspace.yaml in camelCase form.

chore: enforce pnpm via devEngines

8cbb84f

Block accidental npm/yarn install — npm 10.5+ and pnpm 10.x both honor devEngines.packageManager and refuse to run when it doesn't match.

B4nan force-pushed the chore/migrate-website-to-pnpm branch from 49a31eb to 4c76e79 Compare April 16, 2026 08:12

B4nan and others added 4 commits April 16, 2026 11:10

chore: normalize .gitignore — add yarn.lock and .yarn, drop .pnpm-store

a8ecef6

ci: fix release workflows for pnpm migration

099e236

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: sync renovate allowlist with pnpm-workspace exclusions

83b043b

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: replace devEngines with only-allow preinstall

93bad29

devEngines.packageManager breaks pnpm commands that delegate to npm (pnpm version, pnpm pkg set, etc). Use the battle-tested only-allow approach instead (same as Vite, Vue, Astro).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(website): migrate to pnpm and enable minimum release age#1845

chore(website): migrate to pnpm and enable minimum release age#1845
B4nan wants to merge 11 commits intomasterfrom
chore/migrate-website-to-pnpm

B4nan commented Apr 14, 2026

Uh oh!

codecov bot commented Apr 14, 2026 •

edited

Loading

Uh oh!

Pijukatel left a comment •

edited

Loading

Uh oh!

Pijukatel left a comment •

edited

Loading

Uh oh!

B4nan commented Apr 15, 2026

Uh oh!

vdusek left a comment •

edited

Loading

Uh oh!

B4nan commented Apr 16, 2026

Uh oh!

vdusek commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

B4nan commented Apr 14, 2026

Summary

Changes

Uh oh!

codecov bot commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Pijukatel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Pijukatel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

B4nan commented Apr 15, 2026

Uh oh!

vdusek left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

B4nan commented Apr 16, 2026

Uh oh!

vdusek commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Apr 14, 2026 •

edited

Loading

Pijukatel left a comment •

edited

Loading

Pijukatel left a comment •

edited

Loading

vdusek left a comment •

edited

Loading